Cloud Storage Setup Guides

In this topic, the available Cloud Storage configurations are explained :

when setting up a Medium on the Global tab of the Admin Configuration.
when configuring an Import Task, Export Task, Data Import or Data Export, in the Data Exchange chapter.

The available Cloud Storage configurations are :

Azure Blob storage
- Using Shared Access
- Using Access keys

AWS S3 storage
- Get AWS S3 credentials
- Configure Amazon S3 storage for a task

Google Cloud Storage

Get Google Storage credentials
Configure Google Storage for a task

Azure Blob storage

Using Shared Access

Go to the Azure Portal.

Go to https://portal.azure.com/ and sign in with an Azure account.

Go to the Azure services.

From the list of available Azure services, look for the "Storage Accounts".

Access the allowed Storage account.

Collect the "container".

Go to the "Containers" and take note of which container that is going to be used.

Get to the "Shared access signature".

In the menu, under "Security + networking", you can click the "Shared Access signature". Here you can set specific properties depending on the needs, such as IP filtering and expiration times. Make sure that 'Blob' is selected as allowed services and 'Container' as allowed resource type:

6. Once the configuration is complete, click 'Generate SAS and connection string'. The generated connection can then be used.

Using Access keys

For the Azure BLOB Cloud storage setup the following information is needed:

Connection string

Container name

Get Blob credentials

Prerequisite: A "Storage Account" has been setup.
The following tutorial from Microsoft explains how to setup a storage account: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal

Go to the Azure Portal.

Go to https://portal.azure.com/ and sign in with an Azure account.

Go to the Azure services.

From the list of available Azure services, look for the "Storage Accounts".

Access the allowed Storage account.

Collect the "container".

Go to the "Containers" and take note of which container that is going to be used.

Get to the "access keys".

In the menu, under "Security + networking", you can click the "Access keys".

Copy one of the "connection string" fields to use it in the configuration of your tasks.

You can "Show/Hide keys" to reveal the values in a safe way.
"Rotate key" will generate a new "key" & "Connection string", in case you fear that the current keys are exposed.

Configure Azure Blob storage for a task

Depending on what you are configuring: a Task or a Medium (in Admin Config).

Select "Azure Blob Storage" as "Medium" and complete the requested fields:

Name — The internal name for easy usage in Marigold Engage.
Description — Additional information on the medium.
API name — A unique API name for configuring the medium through the REST API.
Medium — The desired medium type. Select "Azure Blob Storage".
Connection string — The full connection string like presented in the Azure Portal.
Container name — The name of the storage container like presented in the Azure Portal.
Subfolder — The name of the folder, in the storage container. If not set, the root level will be used.

Medium configuration:

Task configuration:

AWS S3 storage

Note: 'AWS' stands for Amazon Web Services. 'S3' is Simple Storage Service that stores data as objects within buckets.

For the AWS S3 Cloud Storage setup, the following information is needed:

Access Key ID
Secret Access Key
Bucket name
Region Endpoint Code

Get AWS S3 credentials

Prerequisite: An "S3 bucket" has been setup + your AWS IAM user has permissions to access the "Security Credentials" section in the AWS Console.

The following tutorial from Amazon Web Services explains how to setup an S3 bucket: https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html

The details on how to create and manage AWS Access credentials, can be found in the AWS docs: https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html

Go to the AWS Management Console.

Go to https://signin.aws.amazon.com/ and sign in with an Amazon Web Services account.

From the list of available AWS services, look for "Storage" >> "S3".

Go to the Amazon S3 service.

The Amazon S3 service displays all the available S3 buckets:

Get AWS account information.

After you are logged in, you can click "My Security Credentials" in the account information drop-down in the top bar.

On the "Access keys (access key ID and secret access key)" tab, create a new access key.

Configure Amazon S3 storage for a task

Depending on what you are configuring: a Task or a Medium (in Admin Config).

Select "Amazon S3 Storage" as "Medium" and complete the requested fields:

Name — The internal name for easy usage in Marigold Engage.
Description — Additional information on the medium.
API name — A unique API name for configuring the medium through the REST API.
Medium — The desired medium type. Select "Amazon S3 Storage".
Access Key ID — This refers to the AWS Access key ID, like presented in the "AWS account information" part.
Secret Access Key — This refers to the AWS Secret Access Key, like presented in the "AWS account information" part.
Bucket name — The name of the S3 bucket storage like presented in the AWS Console >> Services >> S3.
Region Endpoint Code — The region endpoint code that refers to the entry point for an AWS web service (see https://docs.aws.amazon.com/general/latest/gr/rande.html for the full list of Amazon Simple Storage Service endpoints).
Subfolder — The name of the folder, in the S3 bucket. If not set, the root level will be used.

Medium configuration:

Task configuration:

Google Cloud Storage

For the Google Cloud Storage setup the following information is needed:

Service account JSON key

Bucket name

Get Google Storage credentials

Prerequisite: A "Google Service Account" & "Google bucket" have been setup.

The following tutorial from Google explains how to setup a Google Service account: https://cloud.google.com/iam/docs/creating-managing-service-account-keys

The Google documentation describes all the steps to create buckets: https://support.google.com/cloud/answer/6250993?hl=en

Go to the Google Console

Manage your service account.

In the Cloud Console, go to the Service Accounts page: https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts

The generated JSON key will be similar to the following sample JSON key:

This JSON key can be used in the "mediums" configuration.

Note: As the private_key can be very long, you can copy it to your clipboard and paste it into your Medium or Task configuration.

Navigate to the Storage.

In the Navigation menu, select "Cloud Storage" to go to your Storage browser page: https://console.cloud.google.com/storage/browser

The bucket names displayed in this overview, are the bucket names that can be used in the "mediums" configuration.

Configure Google Storage for a task

Depending on what you are configuring: a Task or a Medium (in Admin Config).

All the required Google Storage information is provided on the "Service Accounts" page.

Select "Google Cloud Storage" as "Medium" and complete the requested fields:

Name — The internal name for easy usage in Marigold Engage.
Description — Additional information on the medium.
API name — A unique API name for configuring the medium through the REST API.
Medium — The desired medium type. Select "Google Cloud Storage".
project_id — The Google Cloud project ID, for which the Service Account is configured.
private_key_id — The id of the private key, provided by the Service Account in the Google Console.
private_key — The pem encoded RSA private key, provided by the Service Account in the Google Console.
client_email — The email address of the service account, provided by the Service Account in the Google Console.
client_id — The id of the service account in Google IAM, provided by the Service Account in the Google Console.
auth_uri — The authorization server endpoint URI, provided by the Service Account in the Google Console.
token_uri — The token server endpoint URI, provided by the Service Account in the Google Console.
auth_provider_x509_cert_url — The URL of the public x509 certificate, used to verify the signature on JWTs, such as ID tokens, signed by the authentication provider, provided by the Service Account in the Google Console.
client_x509_cert_url — The URL of the public x509 certificate, used to verify JWTs signed by the client, provided by the Service Account in the Google Console.
Subfolder — The name of the folder, in the storage container. If not set, the root level will be used